method of registering multimodal data images using 3D geographic arcs and an aerial vehicle.

专利摘要:
RECORDING MULTIMODAL DATA IMAGES USING 3D GEOGRAPHIC ARCHES. The present invention relates to an accurate, flexible and scalable technique for recording multimodal images, a technique that does not need to rely on the direct combination of resources and does not need to rely on precise geometric models. The methods and / or systems described here allow the recording (fusion) of multimodal images of a scenario (700) with a three-dimensional (3D) representation of the same scenario (700) using, among other information, data from the point of view of a sensor (706, 1214, 1306) that hurt a target image (402), as well as 3D-Geoarcs. The recording techniques of the present description can be comprised of three main steps, as shown in figure 1. The first main step includes forming a 3D reference model of a scenario (700). The second main step includes estimating the 3D geospatial point of view of a sensor (706, 1214, 1306) that generates a target image (402) using 3D-GeoArcs. The third main step includes projecting the target image data into a 3D composite scenario representation
公开号:BR102013010781B1
申请号:R102013010781-6
申请日:2013-04-30
公开日:2021-01-26
发明作者:Yuri Owechko
申请人:The Boeing Company；
IPC主号:

专利说明:

BACKGROUND
[0001] The present description refers to image recording systems and methods, and more particularly to image recording techniques that combine one or more multimodal images of a scenario with a 3D representation of the same scenario using 3D geographic arcs (3D -GeoArcs ”.
[0002] Image registration is the process of combining one or more sets of data to form a single representation of data. The data sets can be multiple photographs, data from different sensors, data from different times, or data stored in different formats. Image recording techniques may involve combining two or more images, or selected points of the images, to produce a composite image containing data from each original image. Some image recording techniques can project details from one data set (referred to as the destination) onto a second data set (referred to as the reference). Some image registration techniques can compare or transform a target image to align with one or more stored reference images. These image recording techniques can use algorithms to relate points between images so that the related points or structures in the images are correlated in the resulting composite image.
[0003] Some methods of registering images seek detailed correspondences between particular resources in the images, such as points, lines and contours that look similar. These appearance-based methods use discovered matches to transform or map a target image to one or more reference images. These techniques may involve the entry of individuals with knowledge of the type of object, scenery or structure represented in the images. Knowledge can identify a set of reference point features in the images that the recording technique should try to correlate. For example, an imaging method can compare two MRI images of different axial parts of a human head, and a doctor can identify points (and / or contours that surround these points) that correspond to the cerebellum (a reference point) in two images. The image registration algorithm can then map the target image to the reference image with a known relationship between the reference points. Thus, by combining landmarks visible in the target image with landmarks previously identified in the reference image, the recording technique can draw conclusions about how the target image aligns with the reference image.
[0004] Other image recording methods compare detailed volume-based (or 3D geometric) images to images using the correlation metric. Some of these geometry-based methods can then measure a distance that represents the disparity between a target image and a reference image based on how closely the volume-based features align. The registration of the two images can use an optimization equation that helps to find a mapping that reduces this distance measurement. These methods can register whole images or sub-images, and if sub-images are registered, the sub-images can be treated as corresponding resource points.
[0005] Some other methods of recording images used geospatial information to provide a reference source for resource structures. Geospatial information (or geographic location) generally refers to the identification of an object's actual geographic location. Geographic location may refer to the practice of assessing the location, or the actual assessed location. Geospatial information can indicate links between resources in photographs and the actual geographic location of those resources or structures. For example, some current location-recognition algorithms use collections of user-provided GPS-tagged images from online repositories in conjunction with direct feature combining and multi-view geometry techniques. A target image can be obtained from a corner or a monument and then the location recognition algorithm tries to find the most similar feature in a reference database by analyzing a large number of saved reference images. These location recognition algorithms require large-scale image databases to allow this geographic location. SUMMARY
[0006] One or more modalities of the present description describe methods, systems, techniques and / or solutions that allow the registration (fusion) of multimodal images of a scenario with a three-dimensional (3D) representation of the same scenario using, among other information, data from the point of view of a sensor that generates a target image, as well as 3D-GeoArcs.
[0007] In an advantageous embodiment, a method for registering images can be performed (for example, at least in part on a data processing system such as a computer), in which the method comprises the following steps. The first stage of the method may include establishing a three-dimensional reference model for a scenario. The next step in the method may include acquiring a target image of the scenario, possibly the target image is captured with a sensor. The next step in the method may include determining the point of view of the sensor that captured the target image using one or more three-dimensional geographic arcs (also referred to as GeoArcs), where the point of view of the sensor can be determined in relation to the model of three-dimensional reference. The next step in the method may include generating a three-dimensional representation composed of the scenario by associating the data from the target image with the data from the three-dimensional reference model, in which the sensor's point of view can be used to perform the association.
[0008] In some embodiments of the present description, the step of determining the point of view of the sensor may additionally comprise the following sub-steps. The first sub-step may include identifying one or more pairs of resources in the three-dimensional reference model. The three-dimensional reference model can be established using information from a geospatial intelligence system database. The next sub-step may include identifying one or more pairs of resources in a target image. The resource pairs identified in the three-dimensional reference model and the target image can be characterized so that they are invariant with the rotation and scale of the reference model and the target image. Then, for each pair of resources in the target image, the method can (1) associate the pair of resources in the target image with one of the pairs of resources in the three-dimensional reference model; (2) estimate an angle associated with the pair of resources in the target image; (3) generate a three-dimensional geographic arc surface associated with the three-dimensional reference model, in which the geographic arc surface can represent the relationships between the pair of resources in the target image and the estimated angle. The next sub-step may include identifying locations in the three-dimensional space relative to the three-dimensional reference model where two or more three-dimensional geographic arc surfaces overlap. In some embodiments of the present description, generating a three-dimensional geographic arc surface may include representing uncertainty in the estimated angle by varying the thickness of the geographic arc surface. In some embodiments of the present description, the generated geographic arc surface can overlap a previously generated geographic arc surface, creating a three-dimensional volume.
[0009] In some embodiments of the present description, the step of determining the point of view of the sensor may additionally comprise the following sub-steps. The first sub-step may include selecting how the sensor's viewpoint determines a location where most geographic arc surfaces overlap. Then, for each pair of features in the target image, the method can refine the generated three-dimensional geographic arc surface by ignoring or removing portions of the three-dimensional geographic arc surface that refer to views that are incorrect based on checks with reference data. The next sub-step may include validating the point of view of the given sensor when referring to reference data to provide for additional features that should be visible in the target image if the point of view of the given sensor is correct. In some embodiments of the present description, the step of determining the point of view of the sensor may include dividing the three-dimensional reference model into a number of regions and determining a point of view of the potential sensor within one or more regions.
[00010] In some embodiments of the present description, the step of generating a composite three-dimensional representation may further comprise the following sub-steps. The first sub-step may include determining the location of the sensor and the angle of the sensor relative to the composite three-dimensional representation. The next sub-step may include determining the location of one or more visible objects in the target image in relation to the composite three-dimensional representation, for each object, by adding a translational offset to the sensor location. The next sub-step may include projecting in real time one or more objects associated with the target image in the 3D composite scenario.
[00011] In another advantageous embodiment, a method for registering images can be performed (for example, at least in part in a data processing system such as a computer), in which the method comprises the following steps. The first step in the method may include identifying one or more pairs of resources in a three-dimensional reference model. The next step in the method may include identifying one or more pairs of resources in a target image. Then, for each pair of resources in the target image, the method can (1) associate the pair of resources in the target image with one of the pairs of resources in the three-dimensional reference model; (2) estimate an angle associated with the pair of resources in the target image; (3) generate a three-dimensional geographic arc surface associated with the three-dimensional reference model, in which the geographic arc surface can represent relationships between the pair of resources in the target image and the estimated angle; and (3) identify the locations in the three-dimensional space in relation to the three-dimensional reference model where two or more three-dimensional geographic arc surfaces overlap. In some embodiments of the present description, the uncertainty in the estimated angle can be represented by varying the thickness of the geographic arc surface. In some embodiments of the present description, the generated geographic arc surface can overlap a previously generated geographic arc surface, creating a three-dimensional volume.
[00012] In another advantageous embodiment, an aerial vehicle is described, comprising a sensor adapted to capture images, and a data processing system coupled communicatively to the sensor. The data processing system can be programmed to establish a three-dimensional reference model for a scenario. The data processing system can be additionally programmed to acquire a target image of the sensor scenario. The data processing system can be additionally programmed to determine the point of view of the sensor that captured the target image using one or more three-dimensional geographic arcs, where the point of view of the sensor can be determined in relation to the three-dimensional reference model. . The data processing system can be additionally programmed to generate a three-dimensional representation composed of the scenario by associating the target image data with data from the three-dimensional reference model, in which the sensor's point of view can be used to perform the association.
[00013] In some embodiments of the present description, to determine the point of view of the sensor, the data processing system can be additionally programmed to identify one or more pairs of resources in the three-dimensional reference model. The data processing system can be additionally programmed to identify one or more pairs of resources in a target image. For each pair of resources in the target image, the data processing system can be additionally programmed to (1) associate the pair of resources in the destination image with one of the pairs of resources in the three-dimensional reference model; (2) estimate an angle associated with the pair of resources in the target image; (3) generate a three-dimensional geographic arc surface associated with the three-dimensional reference model, where the geographic arc surface can represent the relationships between the pair of features in the target image and the estimated angle. The data processing system can be additionally programmed to identify the locations in the three-dimensional space in relation to the three-dimensional reference model where two or more three-dimensional geographic arc surfaces overlap. In some embodiments of the present description, to generate a three-dimensional representation composed of the scenario, the data processing system can be additionally programmed to project in real time one or more objects associated with the target image in the 3D composite scenario.
[00014] In some embodiments of the present description, the sensor can be located in the aerial vehicle so that landscapes and scenes can be included within the sensor's field of view. In some embodiments of the present description, the data processing system may include a memory, in which the memory can store reference data, including the three-dimensional reference model, and in which the memory can store images captured by the sensor.
[00015] The features, functions and advantages that have been discussed can be independently obtained in various modalities or can be combined in still other modalities whose additional details can be observed with reference to the description and drawings below. BRIEF DESCRIPTION OF THE DRAWINGS
[00016] Various features and advantages are described in the description below, in which various modalities are explained, using the following drawings as examples.
[00017] Figure 1 is an illustration of a high-level flowchart that shows exemplary steps performed according to one or more modalities of the present description.
[00018] Figure 2 is an illustration of a high level flowchart that shows exemplary steps performed according to one or more modalities of the present description.
[00019] Figure 3 shows illustrations of exemplary geometric shapes that can help to describe one or more more modalities of the present description.
[00020] Figure 4 shows an exemplary flow of information that can help to describe one or more modalities of the present description.
[00021] Figure 5 shows illustrations of exemplary geometric shapes that can help to describe one or more modalities of the present description.
[00022] Figure 6 shows illustrations of exemplary geometric shapes that can help to describe one or more modalities of the present description.
[00023] Figure 7 shows an exemplary scenario and the exemplary application of techniques of one or more modalities of the present description.
[00024] Figure 8 shows exemplary results from one or more exemplary tests.
[00025] Figure 9 shows exemplary images and results of one or more exemplary simulations.
[00026] Figure 10 shows a top-down angled view of an aerial vehicle in accordance with one or more embodiments of the present description.
[00027] Figure 11 shows a top-down angled view of an aerial vehicle in accordance with one or more embodiments of the present description.
[00028] Figure 12 shows an illustration of a block diagram of an exemplary data processing system according to one or more embodiments of the present description.
[00029] Figure 13 shows an illustration of a block diagram showing exemplary interactions between the program code and other components according to one or more modalities of the present description.
[00030] Figure 14 shows an illustration of a diagram of an exemplary system of data processing systems and components connected to the network according to one or more modalities of the present description. DETAILED DESCRIPTION
[00031] Even though there are techniques for registering images, there are several disadvantages in current techniques. For example, many current techniques try to find correspondences between specific resources to perform direct registration. These techniques may require complex computations due to the high number of data points that may need to be considered to validate the combination of two resources. Some current techniques need to transform the images, for example, between target images obtained from a camera or sensor and the reference images or models. In addition, these techniques require large-scale searches through reference image databases to find matching resources, and complex algorithms that combine a target image with visually similar images or models in the reference databases. These techniques use a wide variety of complex methods to determine similarity, including the "bag-of-words method" (detecting a number of similar features within an image to estimate total similarity). These complex methods can satisfy well-photographed locations and urban areas, but in more rural areas, databases may be insufficiently occupied. Furthermore, finding correspondences between specific resources can be difficult if the resources are generated from records made at different wavelengths or different types of sensors.
[00032] Current geometry-based methods also have disadvantages. These techniques are widely used in urban environments where accurate geometric models can be generated from real structural data (or terrestrial level). The recording techniques then use different points, repeated patterns, and line-grouping similarities to detect 3D alignment between a target image and a stored model. Although these techniques show promising results, they are not suitable for scenarios or arbitrary areas without significant land-level image coverage to form reference models and reference databases. In addition, current techniques can detect and combine sub-regions in images, and these combined sub-regions can confuse the registration algorithm and result in incorrect registration of the total image.
[00033] Furthermore, current techniques may not be able to accurately record images acquired using different modalities (for example, images acquired with different sensors or images captured in different formats or dimensions), because when registering images of different modalities, the assumptions that current techniques rely on may not prove to be true. For example, many conventional image registration techniques assume that the image intensities of corresponding image elements (for example, pixels, voxels and the like) are identical in the images that will be recorded. Additionally, images of different modalities may have different resolutions, dimensions or formats, or these have been recorded from widely varying points of view.
[00034] Current geographic location registration techniques also have limitations. For example, these do not provide an accurate record based on the varied points of view in which a photograph was taken. Additionally, due to the fact that massive databases of images of other information are required, these techniques are limited to regions with large populations such as urban areas or other areas that attract many visitors (so that visitors can provide images to the databases ). Furthermore, the accuracy of techniques that rely on a combination of features or patterns can degrade with changes in lighting, season or weather. These techniques also suffer from scalability problems due to the massive number of reference images required.
[00035] Therefore, a more precise, flexible and scalable technique is required for the registration of multimodal images, a technique that does not need to rely on a direct combination of resources and does not need to rely on precise geometric models. The present description describes a technique, method and / or system to allow the recording (fusion) of multimodal images of a scenario with a three-dimensional (3D) representation of the same scenario using, among other information, data from the point of view of a sensor that generated a target image, as well as GeoArcs (also referred to as geographic arcs).
[00036] Unlike existing point of view registration systems, the solutions described in this description do not need to try to perform a direct combination of resources or find correspondences between target images and highly detailed reference models. The solutions of the present description can be standardized on how humans are located within a geographic location. Studies in ecological psychology have shown that humans are located in unfamiliar environments using a topographic map of the area. Humans do not form complex models of their neighborhood and try to directly combine specific features in these models with stored reference data. In fact, humans follow an iterative evidence collection process in which they consider relationships between a small number of observable generic benchmark resources. These then form a hypothesis and attempt to validate them using reference data. Similarly, the techniques described here involve gathering evidence, generating multiple hypotheses, and then generating an area of final geographic location using the hypothesis that best satisfies multiple constraints imposed by the reference model and perhaps other reference data. This technique avoids the need to solve the problem of strong correspondence from resource to resource which is the basis for current image registration systems. The solutions described here may only need to determine whether a resource in the target image is the same type as the reference model and whether a resource has similar relationships with other resources. As long as the same types of features (for example, based on edges and corners) can be detected in the target image and in the scenario's 3D reference model, the destination and reference can be merged into a composite 3D representation.
[00037] The recording techniques described here allow an accurate and flexible fusion of multimodal data, such as 2D data (for example, EO, 1R, and SAR data) and 3D data (for example, LIDAR data). EO (Electro-optical) sensors are electronic detectors that convert light, or a change in light, into an electronic signal. These are used in many industrial and consumer applications. Infrared (IR) sensors are electronic devices that measure the radiation of infrared light from objects in your field of view. SAR (Synthetic Aperture Radar) is an electromagnetic image sensor generally used in remote sensing applications. A SAR sensor can be mounted on an aircraft or a satellite, and is used to make a high-resolution image of the earth's surface. LIDAR (Light Detection and Direction, also LADAR) is an optical remote sensing technology that can calculate properties of a destination by illuminating the destination with light, usually using pulses from a laser. Sensors of the type previously described can be integrated into the systems and solutions described in that description. One or more sensors of the type described (or other types) can be coupled to a system that incorporates some or all of the solutions described here. For example, a 2D image sensor can be attached to a general purpose computer that includes a processor that can execute computer code, so the computer, as the processor executes computer code, can accept information from the sensor 2D image and continue to implement the solutions of this description.
[00038] Throughout this description, the term fusion refers to the process of combining and registering one or more target images of a scenario with one or more reference images and / or 3D representations of a scenario. The term geographic registration can also be used throughout this description to refer to the registration process where registration takes place between a destination image and a 3D scenario representation that has been combined with geographic locations and real-world structures. In addition, it should be understood that although the descriptions here may refer to images, such as target images created by sensors, the solutions of the present description may also apply to video. Therefore, the present description contemplates a technique, method and / or system to allow the fusion of video of a scenario with a 3D representation of the same scenario. Although, for the sake of clarity, the present description will mainly refer to target images, the solutions here can work with the target video as well. In addition, although this description generally describes the fusion of target images and / or video with a 3D reference representation, it should be understood that the reference representation can also be a 2D image instead of a 3D representation.
[00039] The registration techniques of the present description can be comprised of three main steps, as shown in Figure 1. Figure 1 is an illustration of a high-level flowchart 100 showing the main exemplary steps carried out according to one or more the modalities of this description. The first main step is a reference model step 102 that includes forming a 3D reference model of a scenario. The second main step is a viewpoint determination step 104 which includes estimating the 3D geospatial viewpoint of a sensor that generated a target image. The third main step is a projection step 106 which includes projecting the target image data into a composite 3D scenario representation.
[00040] The first main step of the registration techniques described here is the reference model step 102. This main step can also include two sub-steps. First, a 3D model of a scenario can be created, referred to as a reference model. The information used to create a 3D model can originate, for example, from a reference map that correlates particular features with locations within a 3D space. In one example, the information used to create a 3D model can originate from a Geospatial Intelligence System database, a database that can be maintained by a government agency or some other entity and can include a comprehensive collection of images , resources, and elevation data related to a variety of locations on Earth. The Geospatial Intelligence System database can correlate particular real-world resources with their geographic locations.
[00041] In the next sub-step, the locations of resources (or reference points) of several in the 3D reference model can be identified and marked as ideal candidates for combining a target image with the 3D reference model. To deal with large potential variations in the point of view from which a target image can be captured, these resources can be characterized or stored so that they are invariant with the image rotation and scale. For example, the solutions described here may just need to determine whether a feature in the reference model is of the same type (ie, similar curves and / or corners) as the target image and whether a feature has similar relationships with other features.
[00042] The second main step of the recording techniques described here is the step of determining point of view 104 which includes estimating the point of view of a sensor that generated a target image. Determining a sensor's point of view may include determining the geographic location information of the sensor (the location of the sensor in a predefined 3D reference scenario) as well as determining the viewing direction (angle) of the sensor. More details can be seen in Figure 2, this is an illustration of a high level flowchart 200 that shows the exemplary substeps carried out according to an example determination step. In this example, the viewpoint determination step can include six sub-steps: (1) sub-step 202 of providing a target image, detecting features in the target image that relate to the types of features identified in the 3D model ; (2) Sub-step 201 of determining angles between pairs of resources in the target image using the calibrated field of view of the sensor; (3) Sub-step 206 for each pair of resources in the 3D model to form pairs of the same types as the target image, generate a 3D-GeoArc (described below) using the angles of pairs of measured target image resources and uncertainties ; (4) Sub-step 208 of refining the 3D-GeoArc results by dividing the portions of each 3D-GeoArc that represent the point of view locations that are not compatible with the reference data; (5) Sub-step 210 of finding the 3D volumes related to the 3D reference model space where most of the 3DGeoArcs overlap (geometric weighting), thus determining an approximate location point of view that is more compatible with the test; and (6) Sub-step 212 of registering the target image with the 3D model and, optionally, validating the registration.
[00043] With reference to the first sub-step 202, to detect the resources in the destination image, it is first assumed that a destination image is obtained by a camera or a sensor at some point. With reference to Figure 1 momentarily, it can be observed that at some point a target image must be generated (task 108), although in some modalities of the present description, the image registration process can be flexible as the precise time in which task 108 occurs. The target image can be captured in real time or main step 104 can use a pre-captured target image. Again with reference to Figure 2, given the destination image, sub-step 202 includes detecting the resources in the destination image. Sub-step 202 solutions may only need to determine whether a resource in the target image is the same type as a resource in the reference model. For example, as long as a feature in the target image has the same type of feature (such as similar edges and / or corners), the feature can be detected in the target image and its spatial relationship to other features is verified with similar feature relationships. in the reference model. This technique avoids the need to solve the problem of strong correspondence from resource to resource which is the basis for current image registration systems. While this same type of resource matching technique may be less accurate than resource-to-resource matching methods, uncertainty in the combination can be addressed in other steps of the registration process.
[00044] In some embodiments of the present description, the reference model can be divided into a number of regions or cells and the detection of resource (sub-step 202) can be carried out separately within each region or cell. One reason for this division may be that the number of resource relationships that will be considered within a total reference model is potentially combinatorial in nature. For example, considering the total reference model, if there are N landscape features in the reference model, and the features are visible in the target image, then a recording technique may need to develop hypotheses in all subsets of element r C (N, r) = N! / (Nr)! R! these resources, and rank them for location consistency. This can provide the best possible localization results, but this technique can be inefficient.
[00045] Some modalities use a hierarchical approach to avoid considering all feature / point of view combinations at once, thereby limiting the potential for a combinatorial explosion in sensor position hypotheses. An exemplary hierarchical approach can first carry out an indeterminate location step through which larger, more distinct pairs of resources are used to divide global research into indeterminate regions. An undetermined region can be characterized by high confidence that the point of view is located within the region, but low precision to verify where the point of view within the indeterminate region is located, perhaps due to the low number of distinct resources and the size of these resources . Examples of a large and distinctive feature that can be used for indeterminate location are mountain tops and large buildings.
[00046] Within each indeterminate region, a hierarchical approach can accomplish a satisfactory localization step and thus smaller resource pairs that are compatible with the larger resource pairs are used to increase the accuracy of the localization point of view. The precise location steps can proceed in order of quality of indeterminate location of indeterminate regions. In addition, these precise regions can be limited based on visibility restrictions. For example, even though the precise location step may consider some minor features, not all features may be visible due to obstructions and the like. In addition, a hierarchical approach can consider the "visible radius" of one or more types of resources and then divide the reference model into cells that correlate with the visibility of one or more resources. The visible radius can be the maximum distance that the sensor can be located from a resource and still be able to capture it. The shape of a cell can be circular, for example, to combine a radius of visibility more precisely, or it can have a different shape (a square, for example). Different cells that correlate with the visibility of different resources can overlap.
[00047] When referring to precise regions or visibility cells, or a combination of these, the recording algorithms and techniques described here can be performed in each region and / or cell individually. Within each region and / or cell, the number of possible resources is limited and the number of potential assumptions can be significantly reduced. For example, in a case where there are K cells, a hierarchical approach could reduce the asymptotic complexity of the algorithm for KC (N / K, r), which is a major improvement. Even though dealing with multiple reference model cells and / or regions may initially result in a greater number of possible end points of view, this uncertainty is dealt with as explained earlier, and will still be dealt with in subsequent sub-steps.
[00048] To understand the rest of the sub-steps (sub-steps 204212), a discussion of GeoArcs (also referred to as Geographic Location Arcs or geographic arcs) can be useful, as GeoArcs can be used to define the relationship (also referred to as a hypothesis) between resource pairs. A description of 2DGeoArcs will be performed first. Figure 3 shows illustrations of geometric shapes that can help describe GeoArcs. Figure 3A includes two points 302, 304 that can represent pairs of resources. Throughout this description, two features in an image that are used to create a GeoArc or a hypothesis can be referred to as a feature pair. Figure 3A also includes multiple points 306, 308, 310 (points of view) of varying distances from the two resources (points 302, 304) and two connecting lines that connect each point of view to the two resources. It can be seen that as the point of view moves away from the two features, the angle (angles 312, 314, 316 are shown for points 306, 308, 310 respectively) between the two connecting lines associated with the point of view decreases. In addition, as shown in Figure 3B, for a given angle between the two connecting lines, there are infinite points of view (located along a circular arc) that can allow the connecting lines at a given angle. It should be understood, in reference to the previous description, that the term "connecting line" does not mean a physical line or connection. Preferably, this refers to the geometric idea of an imaginary straight line that can extend between two points.
[00049] With reference to Figures 4A-C, and considering the previous description, it can be seen how the information of a target image (shown in Figure 4A) can be projected on a reference model (shown in Figure 4C) using a 2D GeoArc. It is assumed that a target image is obtained with a sensor or a camera (for example, from a terrain plot). Figure 4A shows an example of target image 402 that may have been obtained by a camera or sensor. When the camera or sensor obtains the 402 image, it can detect the 404 and 406 resources, and can also estimate them, for example, by examining details about the target image, the angle (0) between the imaginary connecting lines ( not shown in Figure 4A) that extend between the sensor (not shown) and resources 404 and 406. Figure 4B also shows, conceptually, how the sensor can estimate the angle (0) between the imaginary connecting lines that extend between the sensor and the 404 and 406 resources.
[00050] In some modalities, to estimate the angle between the resources, the recording techniques can use details about the calibrated field of view (FOV) of the sensor. In some examples, the FOV details associated with the camera may be known ahead of time. FOV details include, for example, the maximum angle / view (width and height) of a scenario that a sensor can capture at a time. For example, information about the camera lens, focal length, sensor size and the like can provide useful FOV details. Alternatively, the FOV of a sensor can be measured. In some modalities, FOV can also be estimated by adjusting its value to maximize geospatial consistency and accuracy as measured by the overlap of GeoArcs generated from the observed data. In other embodiments, the sensor's FOV can be estimated by examining the details of an image after the image is created. Once the maximum angle / view of a sensor is known (in other words, the full extent of the visible range of sensors), then sub-angles can be estimated for objects that are within the maximum visible range of the sensor.
[00051] Assuming that there is a reference image (for example, an overhead view of a terrain graph), that reference image can include a number of features, thus a number of 2D-GeoArcs can be generated based on in resource pairs. Assuming that the resources 404 and 406 detected in the target image 402 can be combined with two reference resources 414 and 416 in the reference image 412 (shown in Figure 4C), the registration technique can compute two circular arcs 418, 420 (a GeoArc) that is mapped on the reference model 412. The two arcs 418, 420 show the possible points of view where the sensor can be located in relation to the reference image when the sensor captures the target image. Thus, a 2D-GeoArc refers to the circular arc (s) (potentially more than one physical arc) of possible 2D locations in a reference image that are compatible with a particular angular relationship between two features in the image destination. All locations in 2D-GeoArc observe the same angle between the two features. The 2D-GeoArc associated with the reference image limits the possible locations from the point of view of the sensor in 2D space.
[00052] Thus, when combining an angular relation between two features detected in the target image with an angular relation of the reference image, and considering the restrictions of the reference image defined by a GeoArcs associated with the detected angle of the destination image, a hypothesis geographic location can be defined. The geographic location hypothesis is a set of locations in the 2D space from which the target image may have been obtained. For example, a hypothesis of geographic location can be represented by the set {(LFri, LFr2), (LFg1, LFq2), 0}, where LFr; is a reference model feature, LFqj is an input image feature, and 0 is the angular separation between LFq, and LFqj.
[00053] It should be understood, throughout this description, that when referring to decisions, estimates and / or computations that are performed by a sensor, these decisions, estimates and / or computations can be performed by equipment, circuitry or code inside the sensor itself, or alternatively by another device that analyzes the image after the sensor captures the image. For example, in some modalities, the set of circuits or equipment inside the sensor can estimate the angle (0) between the imaginary connecting lines. In other modalities, another device, such as a computer program running on a data processing system, can perform this estimate.
[00054] The same process explained above to create a 2D Geo-Arc / hypothesis can be performed for more resource relationships, and thus more GeoArcs / hypotheses can be generated. The 2D-GeoArcs generated by each resource relationship mapping can then overlap (as shown in Figure 5A), generating a Geographic Location Probability Map (GLM). For example, with reference to Figure 5A and assuming perfect 2D-GeoArcs, the GLM 502 can be a point that consists of the intersection of two or more 2DGeoArcs. If the 2D-GeoArcs are not perfect, the GLM can represent an area in the 2D space within which there is a high probability of having the sensor point of view. In that case, additional GeoArcs can (although not necessarily) provide additional evidence of more accurate location, resulting in a smaller GLM area. Thus, the region in 2D space where the majority of 2D-GeoArcs overlap may present the best evidence to determine the Geographic Location Area (GA). The GA is the final location in 2D space that the registration process determines is most likely to be the location where the target image was obtained.
[00055] Any recording technique will likely have to deal with some level of uncertainty, indicating that GeoArcs may not be perfect lines / arcs. For example, a potential source of error in the techniques described may be one where the sensor that captures the target image (or a method that examines a target image later) must approximate the angle between the two features, as it can be difficult to verify the exact distance and orientation between the sensor and the resources. As can be seen in Figure 5B, for example, errors or uncertainties in the sensor data can introduce uncertainties in the 2D-GeoArc generation process. As Figure 5B shows, uncertainties can result in a wider range of possible 2D locations associated with each 2D-GeoArc, which consequently can result in a "thickness" associated with GeoArc when it is mapped in the reference image. Therefore, as shown in Figure 5C, when uncertainty is introduced into the multiple GeoArcs registration process, the intersection of two 2D-GeoArcs can form an intersection area 510 instead of an intersection point as shown in Figure 5A. However, as more resource relationships are analyzed and more GeoArcs are mapped in the reference image, the intersection area can become relatively small and thus the total level of uncertainty regarding GA can be reduced.
[00056] With an understanding of 2D-GeoArcs as a basis, the following will describe how the concept of 2DGeoArc can be extended to 3D-GeoArcs. Figure 6 shows an angular separation (0) between a pair of resources 602, 604 that exists in 3D space. As shown in Figure 6, a 3D-GeoArc is a surface formed by "dragging" or rotating a 2D-GeoArc (in a plane that contains the feature pair) around the line (or geometric axis) that connects the feature pair . All views on the 3D surface observe the same angle between the two features. Consequently, similar to the concept of 2D-GeoArc, the observed angular separation between a pair of resources in a target image can define a 3D-GeoArc associated with a 3D reference model, and 3D-GeoArc limits the possible point locations of sensor view in 3D space. Therefore, for the 3D case, the resource pair relationship / association hypothesis is a correspondence between a pair of features in the target image (for example, a 2D image) and a pair of features in a 3D reference model. This association (hypothesis) defines a surface in 3D space where the real geographic location can be located, denoted as a 3D-GeoArc.
[00057] Figure 7 shows an example of a 3D 700 scenario where the registration techniques using 3D-GeoArcs described here can be useful. As can be seen in Figure 7, two corners 702, 704 of a building may have been identified as a feature pair by a sensor 706 as a target image of the scenario is captured. Then, the sensor can estimate the angle of separation (0) between the resource pair. So, assuming that the features (corners 702, 704) can be related to two similar features in a 3D reference model, and given the estimated angle (0), the recording technique can create a 3D-GeoArc 708 that looks like a " degenerate torus "with a zero torus orifice size. The surface of this "degenerate torus" can define the range of 3D locations within the 3D reference model where the sensor could be located when the target image of the scenario is captured.
[00058] Additionally, with reference to the registration techniques using 3D-GeoArcs described here, additional resource pairs in the target image can be identified, and as additional resource relationships are analyzed between the destination image and the 3D reference model, additional hypotheses / 3D-GeoArcs can be mapped to the 3D reference model. The 3D-GeoArcs overlap or intersection creates 3D Geographic Location Probability Maps (3D-GLMs). Assuming that there is no uncertainty in the angles of feature pairs, the intersection of two 3D-GeoArcs results in a curved line in 3D space, and the intersection of three GeoArcs results in a point. However, in reality, the 3D-GeoArc record may have to deal with uncertainties that can result from errors or uncertainties in the target image sensors (or devices that analyze a target image later). In relation to 3D-GeoArcs, these uncertainties can "confuse" a GeoArc surface, this results in an intersecting volume of 3DGeoArc if two or more 3D-GeoArcs intersect or overlap. Thus, a 3D GLM can represent a volume in 3D space within which there is a great possibility of having the point of view of the sensor. Additional 3D-GeoArcs can (although not necessarily) provide additional evidence for a more accurate location, resulting in smaller 3D GLM areas. Thus, the volume in 3D space where several 3D-GeoArcs overlap can be a good candidate for the 3D Geographic Location Volume (3D-GV). GV 3D is the final set of locations in 3D space that the registration process determines as the most likely to be the location where the target image was obtained.
[00059] As a sub-step for point-of-view determination step 104 (see Figure 1), each 3D-GeoArc can undergo a refinement process after the 3D-GeoArc is generated and before the 3D- GeoArc be added to the collection of 3D-GeoArcs that can be considered in the weighting sub-step. This may be the case where for a particular generated 3D-GeoArc, it could be inefficient (or erroneous) to consider points along the total GeoArc as possible points of view. Therefore, the refinement process can divide or ignore portions of a 3D-GeoArc that represent point-of-view locations from which the feature pair is not visible due to occlusions (obstructions), in other words, point-of-view locations. from which important resources cannot be detected. In addition, the refinement process can split or ignore portions of a 3D-GeoArc that represent point-of-view locations that are not physically possible based on reference data (for example, data that originates from a geospatial database). For example, if there is an object (such as a large rock or building) where a potential point of view is deduced, that point of view may be indifferent. Also, any other information about where the sensor should (or not) be can help to refine the 3D-GeoArcs.
[00060] Additionally, in some modalities, as explained above, the reference model is divided into a number of overlapping cells or regions. Considering only a small number of resource relationships at a time in multiple regions can result in a relatively large number of possible locations from the initial point of view. However, the candidate group can be refined (many candidates can be rejected) by carrying out checks with baseline data to predict resources that should be visible given a candidate hypothesis. If the predicted resources are observed, then these serve as additional evidence to reinforce the hypothesis and allow the prediction of additional benchmark resources. By proceeding to reduce the number of possible locations through multiple iterations of these refinement techniques, the geographic location can be quickly and efficiently estimated without having to perform a global resource combination.
[00061] Once a refined GeoArcs group has been added to the GeoArcs collection that can be considered, the next sub-step can be performed - selecting or "pondering" the most likely geographic location (3D volume) created by overlaying the 3DGeoArcs . This sub-step looks for the volume of geographic location that is most compatible with the resource pair relationships in the target data and in the scenario's 3D reference model. Each feature pair relationship (3D-GeoArc) imposes geometric constraints on the possible set of sensor views that is compatible with the relationship. Therefore, the most likely sensor point of view can be determined using 3D-GeoArc weighting (or geometric weighting) to find the point of view that most closely satisfies the restrictions imposed by the feature pair angles observed in the target image and the 3D reference model.
[00062] 3D-GeoArc weighting (or geometric weighting) refers to the process of finding volumes (and ideally a volume) with most 3D-GeoArcs superimposed. Therefore, a level of certainty regarding the point of view of a sensor in 3D space can be determined by finding the 3D volume where most 3D-GeoAres overlap, thus determining an approximate point of view location that is more compatible with the proof. This geometric weighting process adds robustness to the registration process. Despite potential errors in target sensor information, and without a densely occupied resource database (required by current registration techniques), the solutions in this description can still accurately and quickly determine the 3D location of a sensor by counting mainly with readily available geospatial map data, unrestricted ground level views.
[00063] The final sub-step within the main step of estimating the point of view serves to register the target image with the 3D reference model and, optionally, to validate the registration. Once the geographic location of the sensor has been determined, the orientation (angle) of the camera can be determined, for example, by referring to the resources used to generate the GeoArcs related to the final point of view. The final image registration (fused to the target image and the 3D reference model) can be performed by combining resources between the target image and the reference model that correspond to the 3D-GeoArcs that contributed to determine the geographical location of the correct camera view. In some modalities, once the final registration process has been completed, the 3D reference model can be used again for validation by providing for additional features that should be visible if the registration / viewpoint determination is correct.
[00064] The third main step of the recording techniques described here is the projection step, which includes projecting the target image data in a 3D scenario representation composed based on the 3D reference model. Once the sensor or camera has been located (point of view determination), the projection step can determine the geographical location of one or more objects identified in the sensor's field of view (from the target image). Using information from the camera / sensor, information from the 3D reference model and relationship information accumulated during the point of view determination step, the projection step can estimate the spatial position within the 3D composite scenario of any pixel in the target image.
[00065] In one example, terrestrial plane information is acquired from the 3D reference model. A ground plane can refer to a 3D surface that is more compatible with the ground in the 3D reference model. Then, similar to the way in which angles can be estimated between resources in a target image, a sensor can estimate the angle associated with a line of sight between the sensor and a pixel in the target image. Then, the projection step can perform a ray tracing technique through which an imaginary line (or a ray) is extended from the point of view / location of the sensor (which can be known) to the ground plane. In this example, each pixel associated with an object on a ground surface can be projected onto a ground plane of the reference model. In another example, a beam can extend until it connects with a building, a bridge or the like, instead of the ground, so that the pixels associated with objects in a structure can be projected. In yet another example, a sensor can estimate the angle associated with a line of sight between the sensor and a pixel in the target image as well as the distance between the pixel and the sensor. Then, the geographic location of the pixel in the 3D composite scenario can be computed as the geographic location of the camera / sensor (the origin) plus a translational deviation (an angle and distance relative to the origin based on angle and distance information estimated by the sensor) .
[00066] The models or avatars of objects detected in the camera can be projected in the correct locations on the map or 3D model. In this respect, the 3D reference model can serve as a common framework for geographically combining and recording 2D multimodal images and also video to create a 3D composite scenario representation. This projection can be made in real time (dynamically) so that the objects identified in the target image can be quickly projected onto the 3D composite scenario, thus providing a common dynamic 3D real-time operational image of a scenario and a fusion structure for combine data from multimodal images. The moving objects detected in the 2D scenery images can be projected onto the 3D structure as real-time avatars.
[00067] It should be understood that even though the descriptions here may refer to target image data that are "projected on the reference model", this phrase should not necessarily be interpreted literally. In some embodiments of this description, the 3D composite scenario representation can be a data composition that is based on the 3D reference model even though it is a separate composition of data In these examples, the target image data can be projected or represented in the composite scenario representation.
[00068] Tests to apply 2D-GeoArcs demonstrated the feasibility of the registration techniques described here, and demonstrate robustness despite errors in target image detection and other uncertainties. In one test, topographic map data from a 16 km2 area in Malibu, CA served as the reference model, and a target image was obtained with a closely located 45 ° field of view (FOV) camera. The test assumed that the observable reference point features were randomly distributed within the FOV camera and that the camera has a minimum (near) and maximum (distant) range within which the features can be detected. For all analyzes, the test assumed 20% error as a measure of separation of angular resource from the estimated FOV. The Geoposition Probability Maps (GLMs) were then calculated by overlaying 2D-GeoArcs for various numbers of landmark features detected in the target image. The Geographic Location Area (GA) was then detected by identifying the portion of the GLM with the highest degree of 2D-GeoArcs overlap. The test includes 50 random exams, with a GA determined for each exam.
[00069] The results of the tests applied to 2D-GeoArcs are shown in Figures 8A-C. Figures 8A-C show three example views of a reference image (for example, terrain), where each reference image includes one or more Probability Geographic Location Maps (GLMs), created as a result of different numbers of landmarks. detected by the camera in the target image. Figure 8A shows the results of a test where the camera detected two reference features 810, 812 and a GLM 802 was created. Figure 8B shows the results of a test where the camera detected three reference point features 814, 816, 818 and two GLMs 804, 806 were created. Figure 8C shows the results of a test where the camera detected four reference features 820, 822, 824, 826 and a GLM 808 was created. In all images in Figures 8A-C, the dark gray regions mark the GLM (s) (s) of possible points where the camera that recorded the target image can be located in relation to the image of reference). As can be seen in the successive images of Figures 8A-C, the GLMs (dark gray regions) rapidly decrease in size as the camera detects more resources. When the camera detects two features (Figure 8A), the GLM 802 is comprised of two crescent shapes (a full 2D-GeoArc). However, when the camera detects four features (Figure 8C), the GLM 808 is comprised of a relatively small area.
[00070] Figures 8D-F show graphs of box and whisker of the GA's (Areas of Final Geographic Location) generated by the 50 random scans as a function of the number of detected resources. Figures 8D-F show three graphs, one for each of the three different distance ranges. Figure 8D shows a graph (the box next) that represents the results where only the reference points at a distance between 2 and 4 km from the camera were considered. (the distant box) that represents the results where only the reference points in a distance between 4 and 6 km from the camera were considered. Figure 8F shows a graph (the mixed box) that represents the results where only the reference points in a distance between 2 and 6 km from the camera was considered. In general, the graphs show that the median GA, denoted by the dark horizontal lines (for example, the line 852, 854, 856, 858, 860), decreases 10X to as the number of resources increases from 3 to 6. For example, in the range of 4 to 6 km (Figure 8E), for 3 resources, the median GA is 0.06 km2, while for 6 resources the median area decreases to 0.005 km2, a reduction 10x. The reduction between maximum and minimum GAs, denoted by the upper and lower parts of light colored boxes (for example, boxes 870, 872) above and below each median GA line, in addition to the 50 tests is even more dramatic, decreases from the highest maximum to the lowest minimum by almost 100x from 3 to 7 resources.
[00071] Additionally, the simulations applied to 3D-GeoArcs demonstrated the feasibility of the registration techniques described here. The test images and results of a simulation example are shown in Figures 9A-C. In this simulation example, a 2D image of a scenario (shown in Figure 9A) was used as a target image. This 2D image was captured using an EO image sensor. Two pairs of resources (902, 904 and 906, 908) were selected in the target image in different orientations. Then, a 3D geographic arc surface (910, 912) was generated for each feature pair (shown in Figure 9B). As can be seen in Figure 9B, the intersection 914 of the two 3D geographic arc surfaces results in a 3D curved line (or a curved volume if uncertainty is introduced). Therefore, the sensor's point of view can be estimated by focusing on the 914 intersection of the 3DGeoArcs, and more 3D-GeoArcs could be considered for additional feature pairs to increase accuracy.
[00072] The simulation also tested a process of creating a 3D reference model that estimates resource types and resource locations when analyzing LIDAR images of the actual scenario (shown in Figure 9C). The process has successfully divided land resource types and resource positions from LIDAR data. Land resources (that is, roads, corners and parking margins) are useful for combining regional position of similar resource types and resource positions between 2D EO images and a 3D representation of the same scenario.
[00073] In addition to the benefits of the registration techniques already described here, the following describes more benefits of one or more modalities. It will be understood that the benefits and advantages described throughout this description are not limitations or requirements, and some modalities may omit one or more of the benefits and / or advantages described.
[00074] One benefit of the solutions in this description is the speed with which complex scenario records can be performed, this allows for scalability. Scalability generally refers to the ability to obtain a technique that works on a small scale and apply it to a problem on a much larger scale. For example, an image recording technique (for example, that uses a direct feature combination) that works well on a simple reference model and a simple target image may not work well in a complex topography, or in an area without a large database of known landmarks, or when changes in lighting make the detected landmarks appear different from the stored landmarks. Since scalability is important for global localization, the authors have developed methods for indexing and combining tens of thousands of visuals, and as a result, these methods also require complex feature trees to search for known features. The indexing of resource trees can be a complex and computationally intensive process, this can directly reduce the storage and combination capacities of the registration process, this can result in reduced distinction of the resource combination.
[00075] The solutions of the present description may not require a direct resource combination and may not need to rely on precise geometric models built from terrestrial level images and 3D mapping. Therefore, the solutions described here may not need to perform combination calculations that are as intensive for the processor as the calculations used by current techniques. Therefore, the solutions described here may be able to compute object geolocation much faster than current techniques, this means that the solutions described here can be scalable to handle much more complicated scenarios. Additionally, due to the fact that the solutions described here do not rely on the combination of direct resource, the geographic location can be computed independently if there is a densely occupied resource database and independent of changes in lighting, season and climate. Also, the solutions described here can use natural and urban resources, this allows generalization to arbitrary environments.
[00076] One or more of the modalities of the present description can have a variety of applications, this can also highlight some benefits of the solutions described here. It will be understood that the applications, benefits and advantages described here are not limitations or requirements, and some modalities may be more suitable for some applications.
[00077] For example, one or more modalities of this description may apply to aerial surveillance and / or UAV's (Unmanned Aerial Vehicles). Figures 10 and 11 show an example scenario 1000, 1100 that includes a UAV 1002, 1102 in flight over a landscape 1004, 1104. Figure 10 shows a top-down angled view of UAV 1002 and Figure 11 shows a view bottom-up angled UAV 1102. Each UAV 1002, 1102 can include a data processing system that can perform some or all of the image recording techniques described here. The details of an example data processing system will be described in more detail below. The data processing system can be mounted on board the UAV, for example, housed within the UAV body. Each UAV 1002, 1102 can also include a sensor or a camera that is capable of capturing an image of the landscape 1004, 1104 within the field of view of the sensor. For example, with reference to Figure 11, a sensor or camera can be located near the tip of the UAV, perhaps at the bottom of the tip, housed within a transparent housing 1106. In this example, due to the fact that the sensor is angled downwards and located at the bottom of the UAV, the sensor can have a wide field of view 1108 (FOV) that allows the sensor to capture large views of the 1104 landscape at once. In operation, the sensor in the UAV can obtain one or more destination images of the landscape below the UAV. Then, a data processing system within the UAV can perform image recording techniques that map details of the target image (s) into a predefined 3D reference model. Alternatively, the UAV can transmit one or more target images to a remote data processing system that can perform recording techniques.
[00078] Aerial vehicles or surveillance locations may require the registration / merging of 2D and 3D multimodal image data from multiple sensors and platforms in a single common operating scenario. In addition to data fusion, sensor models or avatars for objects detected in a sensor field of view may need to be projected in the correct locations on a map or 3D model, providing a dynamic real-time 3D fusion structure and operating scenario ordinary. This image fusion may need to work quickly and correctly over an extended geographic area using many different camera angles. One or more embodiments of the present description can provide such solutions.
[00079] In other examples, one or more of the modalities of the present description can be used for computer vision, medical images, military automatic target recognition, remote sensing (cartography update) and compilation and analysis images and satellite data. Image recording techniques can also be used to record a patient's medical data in an anatomical atlas, such as the Talairach atlas for neuroimaging. Image restoration techniques can also be used in astrophotography to align images obtained from space where a computer uses control points and transforms an image to make larger features align with a second image. Image registration is also an essential part of creating a panoramic image. In addition, there are many different techniques that can be implemented in real time and can be performed on embedded devices such as cameras and camera phones. One or more embodiments of the present description can provide solutions that add flexibility to all of these applications.
[00080] Furthermore, due to the wide applications to which image registration techniques can be applied, it has been difficult until now to develop a general method that is optimized for all uses. In fact, many applications include additional techniques for handling unique application situations. For example, recording medical images of data related to a single patient obtained at different time points generally also involves elastic recording (also known as non-rigid) to deal with the individual's deformation (e.g., deformation due to breathing, anatomical changes, tumor growth, and so on). Because of the flexibility that can be offered by one or more of the modalities of the present description, the present description can provide a general image registration method that is optimized for many, if not all, uses.
[00081] In some implementations of the present description, the techniques, methods, routines and / or solutions described here, including the examples of methods and routines illustrated in one or more flowcharts and block diagrams of the different modalities shown can be performed by a system data processing system that is programmed so that the data processing system is adapted to perform and / or execute the methods, routines and solutions described here. Each block or symbol in a block diagram or flowchart diagram referred to here may represent a module, segment or portion of usable or computer-readable program code that comprises one or more executable instructions to implement, by one or more processing systems. specified function or function. It should be understood that, in some modalities, the function or functions illustrated in the blocks or symbols of a block diagram or flowchart may occur out of the order observed in the figures. For example, in some cases, two blocks or symbols shown in succession can be executed substantially simultaneously or the blocks can sometimes be executed in reverse order depending on the functionality involved. Consequently, the different modalities of the present description can take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides a program code for use by or in conjunction with a computer or any device or system. that executes instructions. Alternatively, the different modalities of the present description may take the form of a computer program stored in (and accessible from) a computer-readable medium such as persistent storage or a hard drive.
[00082] Now with reference to Figure 12, a block diagram of an example data processing system 1200 shown can perform the methods, routines and solutions of the present description. In this example, data processing system 1200 includes a communication matrix 1202 that provides communications between components such as a processor unit 1204, memory 1206, persistent storage 1208, a communication unit 1210, an input / output device (I / O) 1212 and a sensor or camera 1214. A bus system can be used to implement communication matrix 1202 and can be comprised of one or more buses such as a system bus or an input / output bus. The bus system can be implemented using any suitable type of architecture that provides a transfer of data between different components or devices connected to the bus system.
[00083] Processor unit 1204 is for executing instructions (for example, a software program or computer code) that can be loaded into memory 1206 of persistent storage 408 (such as a hard drive) or a computer program product 1220 (such as a CD or DVD). Processor unit 1204 can be a set of one or more processors or it can be a multiprocessor core depending on the particular implementation. In addition, processor unit 1204 can be implemented using one or more heterogeneous processor systems in which a main processor is present with secondary processors on a single chip. As another illustrative example, processor unit 1204 can be a symmetric multiprocessor system containing multiple processors of the same type.
[00084] Memory 1206 in these examples can be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 1208 can take many forms depending on the particular implementation. For example, 1208 persistent storage can contain one or more components or devices. For example, persistent storage 1208 can be a hard disk, a flash memory, a rewritable optical disc, a rewritable magnetic tape or some combination of these. The media used by 1208 persistent storage can also be removable. For example, a removable hard drive can be used.
[00085] The instructions for an operating system may be located in persistent storage 1208. In a specific embodiment, the operating system may be some version of a number of known operating systems. Instructions for applications and / or programs can also be located in persistent storage 1208. These instructions and / or programs can be loaded into memory 1206 for execution by processor unit 1204. For example, the processes of the different modalities described here can be performed by processor unit 1204 using computer-implemented instructions that can be loaded into memory such as memory 1206. These instructions are referred to as program code, computer-usable program code or computer-readable program code that can be read and executed by a processor in processor unit 1204. Program code in different modalities can be incorporated into different physical or tangible computer-readable media such as memory 1206, persistent storage 1208.
[00086] Instructions for applications and / or programs may also be included as part of a 1220 computer program product that is not permanently included in the 1200 data processing system. The 1220 computer program product may include a form of computer readable medium 1222 and program code 1224. For example, program code 1224 can be located in a functional form on computer readable medium 1222 and can be loaded or transferred to data processing system 1200 for execution by the unit processor 1204. Program code 1224 and computer-readable medium 1222 can form a computer-program product 1220. In one example, computer-readable medium 1222 can be in a tangible form, such as an optical disc or magnetic that is inserted or placed in a drive or other device, for transfer to a storage device such as a hard drive that is part of the cabinet persistent watering 1208. The unit or other device may be connected or in communication with other components of the data processing system 1200, for example, through the communication matrix 1202. In another tangible form, the computer-readable medium may be a persistent storage such as a hard disk or flash memory that is connected to the 1200 data processing system.
[00087] For the purposes of this description, a usable computer or computer-readable medium can generally refer to any tangible device that can contain, store, communicate, propagate or transport data (such as a software program) for use by or in together with a system, for example, one that executes instructions. The usable computer or computer-readable medium can be, for example, without limitation, an electromagnetic, optical, electronic magnetic semiconductor or infrared system or a propagation medium. Non-limiting examples of a computer-readable medium include a solid state semiconductor or magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a hard magnetic disk and a optical disc. Optical discs can include compact disc read (CD ROM), compact disc read and write (CD R / W) and DVD. In addition, a computer-usable or computer-readable medium may contain or store a human-readable or computer-usable program code so that when the computer-readable or usable program code is executed on a computer the execution of that human-readable or program code computer usable cause the computer to perform routines, procedures, specified steps and the like. The tangible form of a computer-readable medium is also referred to as a computer-writable storage medium.
[00088] The input / output (I / O) device 1212 allows the input and output of data with other devices that can be connected to the data processing system 1200. For example, the input / output device 1212 can be a CD or DVD drive that can read data stored on a computer-readable medium (such as a CD or DVD), for example, a computer-readable medium 1222. Input / output devices can be coupled to the system directly or via computer controllers. OU stakeholders. Program code 1224 can be transferred to data processing system 1200 from computer-readable medium 1222 via input / output device 1212.
[00089] The different illustrated components of data processing system 1200 are intended to provide architectural limitations to the way in which the different modalities can be implemented. The different illustrative modalities can be implemented in a data processing system including components in addition to or instead of those illustrated for the data processing system 1200. Other components shown in Figure 12 can be varied from the illustrative examples shown.
[00090] Program code 1224 can take the form of computer code that performs many of the computations and processes required by the image registration techniques described here. Figure 13 shows an example of program code 1302 that can include one or more image recording techniques described here, including techniques described by one or more of the flow diagrams in that description. Program code 1302 can be in communication with a reference database 1304, where reference database 1304 contains reference data as one or more reference models required by one or more of the solutions described here. Reference database 1304 can be stored on persistent storage, for example, similar to persistent storage 1208, shown in Figure 12. Alternatively, reference database 1304 can be stored on a storage device that is located remotely at relation to the data processor that can execute program code 1302. Program code 1302 can also be in communication with a camera or sensor 1306, where the sensor may be able to capture one or more target images (for example, a scenario within the field of view of sensor 1308) that will be used by one or more of the solutions described here. Sensor 1306 can be directly coupled to a data processing system, since sensor 1214 in Figure 12 is coupled to data processing system 1200, where the data processing system performs image recording techniques (code program 1302). Alternatively, the system that performs image registration techniques (program code 1302) can be located remotely in relation to sensor 1306, in which case the target images captured by sensor 1306 can be transmitted to the remote system.
[00091] Again with reference to Figure 12, the data processing system 1200 can include a communication unit 1210 which can provide communication between the data processing system 1200 and other data processing systems or devices. In these examples, the communication unit 1210 can be a network interface card. The communication unit 1210 can provide communication through the use of a physical and wireless communication link unit considering data entry. The communication unit can include one or more devices used to transmit and receive data such as a modem or a network adapter. The communication link can be physical or wireless in the illustrative examples. In some embodiments of the present description, if, for example, a reference database or a sensor / camera is installed in a location that is remote from the data processing system 1200, the communication unit 1210 can provide a mode interface that data can be transmitted to and from these remote components. For example, a data processing system 1200 can receive data from a remote sensor / camera via the communication unit 1210, or the data processing system 1200 can receive data from a remote reference database through the communication unit. 1210.
[00092] Figure 14 shows a diagram of an example network system 1400 of data processing systems and components connected to the network according to one or more embodiments of the present description. The network system 1400 may include a network 1410 that potentially connects one or more client machines 1404, potentially one or more server machines, potentially an independent storage device 1406 and a data processing system that can be mounted on board a 1408 vehicle (such as a UAV). The client machine (s) 1404 can, for example, be personal computers or an end user computer. The 1410 network is a means used to provide communication links between various data processing systems (and perhaps other components), and the 1410 network can include connections such as wired or wireless communication links, or perhaps fiber optic cables. The 1410 network can include internet connections and perhaps remote fixed connections. In some examples, the 1410 network represents a worldwide collection of networks and ports that use the suite of protocols such as Transmission Control Protocol (TCP IP) Internet Protocol to communicate. The 1400 networking system can also be implemented as a similar number or different types of networks, for example, an intranet, a local area network (LAN) or an area network (WAN). Typically, each data processing system and component within the network system 1400 could include a communication unit, for example, as the communication unit 410 of Figure 12, thus allowing the data processing system or component to interact with the network 1410 and, in turn, other data processing systems connected by network.
[00093] In one example, a data processing system can be mounted on board a 1408 vehicle (such as a UAV), and the data processing system can include a communication unit that allows wireless communication 1412 with network 1410 and, in turn, other data processing systems connected to the network. The data processing system mounted on vehicle 1408 can be similar to the data processing system 1200 of Figure 12. The data processing system mounted on vehicle 1408 can perform some or all of the recording techniques described here. For example, the data processing system can acquire reference data over the 1410 network (for example, from a server machine 1402 or storage connected to the network 1406), and then perform the rest of the record computation through the data processing system on vehicle 1408. In another example, the data processing system can acquire target images via a sensor on vehicle 1408 and then transmit the target image over network 1410 to a data processing connected to the network (for example, included within a client machine 1404 or a server machine 1402), thus the data processing system connected to the network performs most of the registration calculations.
[00094] Figure 14 is considered as an example and not as an architectural limitation of different modalities. It should be understood that the network system 1400 may include additional server machines (or less) 1402, client machines 1404, storage 1406, vehicles 1408 and perhaps other systems and / or data processing devices not shown. Additionally, vehicle 1408 may be a vehicle other than a UAV, for example, another type of aircraft, automobile, vessel or the like. Additionally, instead of vehicle 1408, or in addition, network system 1400 may include a sensor or camera (and optionally a data processing system) that is contained within a device, but is not a vehicle. For example, a camera system mounted on a pole, building, natural landmark, or even carried by a human.
[00095] In the figures and in the text, in one aspect, a method is described for the registration of images, the method includes establishing a three-dimensional reference model of a scenario (700); acquiring a destination image (402) of the scene (700), the destination image (402) is captured with a sensor (706, 1214, 1306); determine a point of view of the sensor (706, 1214, 1306) that captured the target image (402) using one or more three-dimensional geographic arcs, in which the point of view of the sensor (706, 1214, 1306) is determined in relation to the three-dimensional reference model; and generate a three-dimensional representation composed of the scenario (700) by associating data from the target image (402) with data from the three-dimensional reference model, in which the sensor's point of view (706, 1214, 1306) is used to perform the association . In a variant, the method of registering images includes the step of determining the point of view of the sensor (706, 1214, 1306) which also includes: identifying one or more pairs of resources in the three-dimensional reference model; identify one or more pairs of resources in a target image (402); for each pair of resources in the target image (402), associate the pair of resources in the target image (402) with one of the pairs of resources in the three-dimensional reference model, estimate an angle (0) associated with the pair of resources in the image target (402), generate a three-dimensional geographic arc surface (910, 912) associated with the three-dimensional reference model, where the geographic arc surface (910, 912) represents relationships between the pair of resources in the destination image (402 ) and the estimated angle (0); and to identify the locations in the three-dimensional space relative to the three-dimensional reference model where two or more three-dimensional surfaces of a geographic arc overlap.
[00096] In a variant, the method of registering images includes the step of generating a three-dimensional geographic arc surface (910, 912) which includes representing the uncertainty in the estimated angle (0) by varying the thickness of the geographic arc surface ( 910, 912). In another variant, the image registration method includes the generated geographic arc surface (910, 912) which can overlay a previously generated geographic arc surface (910, 912), creating a three-dimensional volume. In another variant, the method of registering images also includes selecting as the determined point of view of the sensor (706, 1214, 1306) a location where most surfaces of the geographic arc overlap. In yet another variant, the image registration method also includes: for each pair of resources in the target image, refine the generated three-dimensional geographic arc surface (910, 912) by ignoring or removing portions of the three-dimensional geographic arc surface (910 , 912) that relate to points of view that are incorrect based on checks with reference data. In yet another variant, the image recording method also includes: validating the determined point of view of the sensor (706, 1214, 1306) when referring to reference data to provide for additional features that should be visible in the target image (402 ) if the determined point of view of the sensor (706, 1214, 1306) is correct.
[00097] In one example, the image registration method includes the step of generating a composite three-dimensional representation that also includes: determining the location of the sensor (706, 1214, 1306) and the angle (0) of the sensor (706, 1214 , 1306) in relation to the composite three-dimensional representation; and determining the location of one or more visible objects in the target image (402) in relation to the composite three-dimensional representation, for each object, by adding a translational offset to the sensor location (706, 1214, 1306). In yet another example, the image registration method includes the step of generating a three-dimensional representation composed of the scenario (700) which includes projecting in real time one or more objects associated with the target image (402) in the 3D composite scenario (700) . In yet another example, the method of registering images includes establishing a three-dimensional reference model that comprises establishing a three-dimensional reference model using information from a geospatial intelligence system database.
[00098] In one example, the image registration method includes the resources of the resource pairs identified in the three-dimensional reference model and in the target image (402) which are characterized so that they are invariant with the rotation and scale of the image model. reference and destination image (402). In another example, the image recording method includes the step of determining the sensor's point of view (706, 1214, 1306) which includes dividing the three-dimensional reference model into a number of regions and determining a potential sensor point of view (706, 1214, 1306) within one or more regions.
[00099] In one aspect, a method for registering images is described, the method includes: identifying one or more pairs of resources in a three-dimensional reference model; identify one or more pairs of resources in a target image (402); for each pair of resources in the target image (402), associate the pair of resources in the target image (402) with one of the pairs of resources in the three-dimensional reference model, estimate an angle (0) associated with the pair of resources in the image target (402), generate a three-dimensional geographic arc surface (910, 912) associated with the three-dimensional reference model, where the geographic arc surface (910, 912) represents relationships between the pair of resources in the destination image (402 ) and the estimated angle (0); and to identify the locations in the three-dimensional space in relation to the three-dimensional reference model where two or more three-dimensional geographic arc surfaces overlap.
[000100] In a variant, the method of registering images also includes varying the thickness of the geographic arc surface (910, 912) to represent the uncertainty in the estimated angle (0). In another variant, the image recording method includes the generated geographic arc surface (910, 912) that can overlay a previously generated geographic arc surface, creating a three-dimensional volume.
[000101] In one aspect, an aerial vehicle is described including: a sensor (706, 1214, 1306) adapted to capture images; a data processing system communicatively coupled to the sensor (706, 1214, 1306), the data processing system programmed to: establish a three-dimensional reference model of a scenario (700); acquiring a destination image (402) of the scenario (700) from the sensor (706, 1214, 1306); determine a point of view of the sensor (706, 1214, 1306) that captured the target image (402) using one or more three-dimensional geographic arcs, in which the point of view of the sensor (706, 1214, 1306) is determined in relation to the three-dimensional reference model; and generate a three-dimensional representation composed of the scenario (700) by associating data from the target image (402) with data from the three-dimensional reference model, in which the sensor's point of view (706, 1214, 1306) is used to perform the association . In a variant, the aerial vehicle includes determining the point of view of the sensor (706, 1214, 1306), said data processing system is additionally programmed to: identify one or more pairs of resources in the three-dimensional reference model; identify one or more pairs of resources in a target image (402); for each pair of resources in the target image (402), associate the pair of resources in the target image (402) with one of the pairs of resources in the three-dimensional reference model, estimate an angle (0) associated with the pair of resources in the image target (402), generate a three-dimensional geographic arc surface (910, 912) associated with the three-dimensional reference model, where the geographic arc surface (910, 912) represents the relationships between the pair of resources in the destination image ( 402) and the estimated angle (0); and to identify locations in the three-dimensional space in relation to the three-dimensional reference model where two or more three-dimensional geographic arc surfaces (910, 912) overlap.
[000102] In another variant, the aerial vehicle includes generating a three-dimensional representation composed of the scenario (700), said data processing system is additionally programmed to project, in real time, one or more objects associated with the destination image (402 ) in the 3D composite scenario (700). In yet another variant, the aerial vehicle includes the sensor (706, 1214, 1306) which is located in the aerial vehicle so that landscapes and scenarios (700) can be included within a field of view of the sensor. In yet another variant, the aerial vehicle includes the data processing system that includes a memory, in which the memory is operable to store the reference data, including the three-dimensional reference model, and in which the memory is operable to store images. captured by the sensor (706, 1214, 1306).
[000103] The description of the different advantageous modalities has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the modalities in the manner described. Many modifications and variations will become obvious to those skilled in the art. Different advantageous modalities can provide different advantages as compared to other advantageous modalities. The modality or modalities selected are chosen and described to better explain the principles of the modalities, their practical application and to allow other elements versed in the technique to understand the description of various modalities with various modifications appropriate to the particular use contemplated.

权利要求:
Claims (9)
[0001]
1. Image recording method, characterized by the fact that the method comprises: establishing a three-dimensional reference model of a scenario (700); acquiring a destination image (402) of the scene (700), the destination image (402) is captured with a sensor (706, 1214, 1306); determine a point of view of the sensor (706, 1214, 1306) that captured the target image (402) using one or more three-dimensional geographic arcs, in which the point of view of the sensor (706, 1214, 1306) is determined in relation to the three-dimensional reference model; and generate a three-dimensional representation composed of the scenario (700) by associating the data from the target image (402) with the data from the three-dimensional reference model, in which the sensor's point of view (706, 1214, 1306) is used to perform the Association; and where the step of determining the point of view of the sensor (706, 1214, 1306) further comprises: identifying one or more pairs of resources in the three-dimensional reference model; identify one or more pairs of resources in a target image (402); for each pair of resources in the target image (402), associate the pair of resources in the target image (402) with one of the pairs of resources in the three-dimensional reference model, estimate an angle (θ) associated with the pair of resources in the image target (402), generate a three-dimensional geographic arc surface (910, 912) associated with the three-dimensional reference model, where the geographic arc surface (910, 912) represents the relationships between the pair of resources in the destination image ( 402) and the estimated angle (θ); and to identify the locations in the three-dimensional space in relation to the three-dimensional reference model where two or more three-dimensional geographic arc surfaces overlap.
[0002]
2. Image registration method, according to claim 1, characterized by the fact that the step of generating a three-dimensional geographic arc surface (910, 912) includes representing the uncertainty in the estimated angle (θ) when varying the thickness of the geographic arc surface (910, 912); and where the generated geographic arc surface (910, 912) can overlap a previously generated geographic arc surface, creating a three-dimensional volume.
[0003]
3. Image registration method, according to claim 1 or 2, characterized by the fact that it additionally comprises: selecting as the determined point of view of the sensor (706, 1214, 1306) a location where most of the arc surfaces geographic overlap; and validate the determined point of view of the sensor (706, 1214, 1306) when referring to reference data to predict the additional resources that should be visible in the target image (402) if the determined point of view of the sensor (706, 1214, 1306) is correct.
[0004]
4. Image registration method, according to claim 2, characterized by the fact that it additionally comprises: for each pair of resources in the destination image, refine the generated three-dimensional geographic arc surface (910, 912) by ignoring or removing portions of the three-dimensional geographic arc surface (910, 912) that refer to points of view that are incorrect based on checks with reference data.
[0005]
5. Image recording method, according to claim 1, characterized by the fact that the step of generating a composite three-dimensional representation also comprises: determining the location of the sensor (706, 1214, 1306) and the angle (θ) of the sensor (706, 1214, 1306) in relation to the composite three-dimensional representation; and determining a location of one or more objects visible in the target image (402) in relation to the composite three-dimensional representation, for each object, by adding a translational offset to the sensor location (706, 1214, 1306); wherein the step of generating a three-dimensional composite representation of the scenario (700) includes projecting in real time one or more objects associated with the target image (402) in a 3D composite scenario (700); and where establishing a three-dimensional reference model comprises establishing a three-dimensional reference model using information from a geospatial intelligence system database.
[0006]
6. Image registration method, according to claim 2, characterized by the fact that the resources of the resource pairs identified in the three-dimensional reference model and in the target image (402) are configured so that they are invariant with a rotation and scale of the reference model and the target image (402); and where the step of determining the sensor view (706, 1214, 1306) includes dividing the three-dimensional reference model into a number of regions and determining a potential sensor view (706, 1214, 1306) within one or more regions.
[0007]
7. Aerial vehicle characterized by the fact that it comprises: a sensor (706, 1214, 1306) adapted to capture images; a data processing system communicatively coupled to the sensor (706, 1214, 1306), the data processing system programmed to carry out the method as defined in any of the preceding claims.
[0008]
8. Air vehicle, according to claim 7, characterized by the fact that to generate a three-dimensional representation composed of the scenario (700), said data processing system is additionally programmed to project, in real time, one or more objects associated with the target image (402) in a 3D composite scenario (700).
[0009]
9. Aerial vehicle, according to claim 7, characterized by the fact that the sensor (706, 1214, 1306) is located in the aerial vehicle so that landscapes and scenarios (700) can be included within a field of view of the sensor; and where the data processing system includes a memory, and where the memory is operable to store reference data, including the three-dimensional reference model, and where the memory is operable to store images captured by the sensor (706, 1214 , 1306).

类似技术:

---

公开号 | 公开日 | 专利标题

---

BR102013010781B1|2021-01-26|method of registering multimodal data images using 3D geographic arcs and an aerial vehicle.

---

ES2855119T3|2021-09-23|Method and device for real-time mapping and localization

---

US10598489B2|2020-03-24|Visual odometry and pairwise alignment for high definition map creation

---

US9989969B2|2018-06-05|Visual localization within LIDAR maps

---

Bazin et al.2010|Motion estimation by decoupling rotation and translation in catadioptric vision

---

Brenner2009|Extraction of features from mobile laser scanning data for future driver assistance systems

---

Wang et al.2010|Automated road sign inventory system based on stereo vision and tracking

---

US20200401617A1|2020-12-24|Visual positioning system

---

Andersson et al.2018|Simultaneous localization and mapping for vehicles using ORB-SLAM2

---

Gracias2002|Mosaic-based visual navigation for autonomous underwater vehicles

---

Liu et al.2021|Simultaneous Localization and Mapping Related Datasets: A Comprehensive Survey

---

Saeedi et al.2009|Vision-aided inertial navigation for pose estimation of aerial vehicles

---

Wang et al.2016|SLAM-based cooperative calibration for optical sensors array with GPS/IMU aided

---

Liu et al.2018|Absolute orientation and localization estimation from an omnidirectional image

---

Krishnaswamy et al.2008|Sensor fusion for GNSS denied navigation

---

Li et al.2015|Error aware multiple vertical planes based visual localization for mobile robots in urban environments

---

Del Pizzo et al.2018|Reliable vessel attitude estimation by wide angle camera

---

Antigny et al.2018|Fusion of 3D GIS, vision, inertial and magnetic data for improved urban pedestrian navigation and augmented reality applications

---

Meyer et al.2020|Long-Term Visual Localization in Large Scale Urban Environments Exploiting Street Level Imagery

---

Qu2017|Landmark based localization: detection and update of landmarks with uncertainty analysis

---

Berveglieri et al.2015|Locating Control Points In Aerial Images With A Multi‐Scale Approach Based On Terrestrial Image Patches

---

Jarron2020|Wide-angle Lens Camera Calibration using Automatic Target Recognition

---

KR20210098534A|2021-08-10|Methods and systems for creating environmental models for positioning

---

de Aguiar2019|Odometria visual monocular em robôs para a agricultura com camara | com lentes" olho de peixe"

---

Baldoni et al.2022|A Semantic Segmentation-based Approach for Train Positioning

同族专利:

---

公开号 | 公开日

---

EP2660777A3|2016-11-02|

---

JP6168833B2|2017-07-26|

---

AU2013200708A1|2013-11-14|

---

JP2013232195A|2013-11-14|

---

AU2013200708B2|2017-07-20|

---

US8855442B2|2014-10-07|

---

CN103377476A|2013-10-30|

---

EP2660777A2|2013-11-06|

---

US20130287290A1|2013-10-31|

---

EP2660777B1|2019-01-02|

---

CN103377476B|2018-04-03|

---

BR102013010781A2|2015-06-23|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

WO1997036147A1|1996-03-28|1997-10-02|Synthonics Incorporated|Methods and apparatus for using image data to determine camera location and orientation|

---

IL120136D0|1997-02-03|1997-06-10|Yissum Res Dev Co|Synthesizing virtual two dimensional images of three dimensional space from a collection of real two dimensional images|

---

KR20000068660A|1997-07-29|2000-11-25|요트.게.아. 롤페즈|Method of reconstruction of tridimensional scenes and corresponding reconstruction device and decoding system|

---

JP3720587B2|1998-07-13|2005-11-30|大日本印刷株式会社|Image synthesizer|

---

US20010036302A1|1999-12-10|2001-11-01|Miller Michael I.|Method and apparatus for cross modality image registration|

---

EP2357841B1|2002-03-27|2015-07-22|Sanyo Electric Co., Ltd.|Method and apparatus for processing three-dimensional images|

---

CN100410622C|2004-05-14|2008-08-13|佳能株式会社|Information processing method and apparatus for finding position and orientation of targeted object|

---

US7580591B2|2005-07-01|2009-08-25|The Boeing Company|Method for generating a synthetic perspective image|

---

EP1862969A1|2006-06-02|2007-12-05|Eidgenössische Technische Hochschule Zürich|Method and system for generating a representation of a dynamically changing 3D scene|

---

US9123159B2|2007-11-30|2015-09-01|Microsoft Technology Licensing, Llc|Interactive geo-positioning of imagery|

---

US8191828B2|2008-05-13|2012-06-05|The Boeing Company|Methods and apparatus for aerial recovery of flying apparatus|

---

US20090295791A1|2008-05-29|2009-12-03|Microsoft Corporation|Three-dimensional environment created from video|

---

WO2012018497A2|2010-07-25|2012-02-09|Raytheon Company|ENHANCED SITUATIONAL AWARENESS AND TARGETING SYSTEM|US9501700B2|2012-02-15|2016-11-22|Xactware Solutions, Inc.|System and method for construction estimation using aerial images|

---

CA2902040C|2013-02-22|2019-07-23|Ohio University|Reduction of sensor captured data streamed to an operator|

---

AU2014295972B2|2013-08-02|2018-10-11|Xactware Solutions, Inc.|System and method for detecting features in aerial images using disparity mapping and segmentation techniques|

---

CN103810686A|2014-02-27|2014-05-21|苏州大学|Seamless splicing panorama assisting driving system and method|

---

WO2015160292A1|2014-04-14|2015-10-22|Saab Vricon Systems Ab|A target determining method and system|

---

CN104036520B|2014-07-02|2017-02-15|中国科学院长春光学精密机械与物理研究所|Multi-sensor building image registration method|

---

US9904055B2|2014-07-25|2018-02-27|Microsoft Technology Licensing, Llc|Smart placement of virtual objects to stay in the field of view of a head mounted display|

---

US9858720B2|2014-07-25|2018-01-02|Microsoft Technology Licensing, Llc|Three-dimensional mixed-reality viewport|

---

US9865089B2|2014-07-25|2018-01-09|Microsoft Technology Licensing, Llc|Virtual reality environment with real world objects|

---

US10311638B2|2014-07-25|2019-06-04|Microsoft Technology Licensing, Llc|Anti-trip when immersed in a virtual reality environment|

---

US10451875B2|2014-07-25|2019-10-22|Microsoft Technology Licensing, Llc|Smart transparency for virtual objects|

---

US9766460B2|2014-07-25|2017-09-19|Microsoft Technology Licensing, Llc|Ground plane adjustment in a virtual reality environment|

---

US10416760B2|2014-07-25|2019-09-17|Microsoft Technology Licensing, Llc|Gaze-based object placement within a virtual reality environment|

---

US10670396B2|2014-09-08|2020-06-02|Sikorsky Aircraft Corporation|Multi-sensor target location registration|

---

CN104751458B|2015-03-23|2017-08-25|华南理工大学|A kind of demarcation angular-point detection method based on 180 ° of rotation operators|

---

CN104778284B|2015-05-11|2017-11-21|苏州大学|A kind of spatial image querying method and system|

---

FR3037429B1|2015-06-15|2018-09-07|Donecle|SYSTEM AND METHOD FOR AUTOMATIC SURFACE INSPECTION|

---

US9830706B2|2015-09-17|2017-11-28|Skycatch, Inc.|Generating georeference information for aerial images|

---

CN105744222B|2016-02-03|2019-03-26|普宙飞行器科技（深圳）有限公司|A kind of unmanned plane wireless image/transmission of video, shared and storage system|

---

FR3049744B1|2016-04-01|2018-03-30|Thales|METHOD FOR SYNTHETICALLY REPRESENTING ELEMENTS OF INTEREST IN A VISUALIZATION SYSTEM FOR AN AIRCRAFT|

---

SE539807C2|2016-04-06|2017-12-05|Advanced Technical Solutions In Scandinavia Ab|A method for generating a 3d combined model representing a scene|

---

US10013798B2|2016-08-30|2018-07-03|The Boeing Company|3D vehicle localizing using geoarcs|

---

US10402675B2|2016-08-30|2019-09-03|The Boeing Company|2D vehicle localizing using geoarcs|

---

US10482339B2|2016-12-09|2019-11-19|United States Of America As Represented By The Secretary Of The Air Force|Quantifying computer vision algorithm performance in the presence of system uncertainty|

---

US10235566B2|2017-07-21|2019-03-19|Skycatch, Inc.|Determining stockpile volume based on digital aerial images and three-dimensional representations of a site|

---

US10515458B1|2017-09-06|2019-12-24|The United States Of America, As Represented By The Secretary Of The Navy|Image-matching navigation method and apparatus for aerial vehicles|

---

WO2019133374A1|2017-12-29|2019-07-04|Walmart Apollo, Llc|System and method for determining autonomous vehicle location using incremental image analysis|

---

US11055862B2|2018-10-26|2021-07-06|Here Global B.V.|Method, apparatus, and system for generating feature correspondence between image views|

---

CN110472670B|2019-07-24|2022-03-01|上海联影智能医疗科技有限公司|Image centerline detection method, computer device, and storage medium|

---

US11094113B2|2019-12-04|2021-08-17|Geomni, Inc.|Systems and methods for modeling structures using point clouds derived from stereoscopic image pairs|

---

CN113724300A|2020-05-25|2021-11-30|北京达佳互联信息技术有限公司|Image registration method and device, electronic equipment and storage medium|

法律状态:
2015-06-23| B03A| Publication of an application: publication of a patent application or of a certificate of addition of invention|

---

2018-12-04| B06F| Objections, documents and/or translations needed after an examination request according art. 34 industrial property law|

---

2019-11-19| B06U| Preliminary requirement: requests with searches performed by other patent offices: suspension of the patent application procedure|

---

2020-09-29| B09A| Decision: intention to grant|

---

2021-01-26| B16A| Patent or certificate of addition of invention granted|Free format text: PRAZO DE VALIDADE: 20 (VINTE) ANOS CONTADOS A PARTIR DE 30/04/2013, OBSERVADAS AS CONDICOES LEGAIS. |

优先权:

---

申请号 | 申请日 | 专利标题

---

US13/459,643|2012-04-30|

---

US13/459,643|US8855442B2|2012-04-30|2012-04-30|Image registration of multimodal data using 3D-GeoArcs|

---